Lectures/Theme 3 - Model Engineering Pipelines/Lecture 19 - Transformer Networks/Lecture 19 - Transformer Networks.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kPG5INqg-_qn"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/avakanski/Fall-2022-Python-Programming-for-Data-Science/blob/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/Lecture%2019%20-%20Transformer%20Networks.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3JhHIQqNe4Qs"
      },
      "source": [
        "<a name='section0'></a>\n",
        "# Lecture 19 Transformer Networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEkmemKte4Qv"
      },
      "source": [
        "- [19.1 Introduction to Transformers](#section1)\n",
        "- [19.2 Self-attention Mechanism](#section2)\n",
        "- [19.3 Multi-head Attention](#section3)\n",
        "- [19.4 Encoder Block](#section4)\n",
        "- [19.5 Positional Encoding](#section5)\n",
        "- [19.6 Using a Transformer Model for Classification](#section6)\n",
        "- [19.7 Fine-tuning a Pretrained BERT Model](#section7)\n",
        "- [19.8 Decoder Sub-network](#section8)\n",
        "- [19.9 Vision Transformers](#section9)\n",
        "- [References](#section10)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gTFgm_bJe4Qw"
      },
      "source": [
        "<a name='section1'></a>\n",
        "\n",
        "# 19.1 Introduction to Transformers "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFMFGCdwiw5k"
      },
      "source": [
        "***Transformer Neural Networks***, or simply ***Transformers***, is a neural network architecture introduced in 2017 in the now-famous paper [“Attention is all you need”](https://arxiv.org/abs/1706.03762). The title refers to the attention mechanism, which forms the basis for data processing with Transformers.  \n",
        "\n",
        "Transformer Networks have been the predominant type of deep learning models  for NLP in recent years. They replaced Recurrent Neural Networks in all NLP tasks, and also, all Large Language Models employ the Transformer Network architecture. As well as, Transformer Networks were recently adapted for other tasks and have outperformed other machine learning models for image processing and video processing tasks, protein and DNA sequence prediction, time-series data processing, and have been used for reinforcement learning tasks. Consequently, Transformers are currently the most important neural network architecture."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GF5bogcC7mUa"
      },
      "source": [
        "<a name='section2'></a>\n",
        "\n",
        "# 19.2 Self-attention Mechanism"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rWMSWyTqjNym"
      },
      "source": [
        "***Self-attention*** in neural networks is a mechanism that forces a model to attend to portions of the data when making predictions. For instance, in NLP, self-attention mechanism is used to identify words in sentences that have significance for a given (query) word in the sentence. That is, the model should pay more attention to some words in sentences, and less attention to other words in sentences that are less relevant for a given task.  \n",
        "\n",
        "In the following two sentences, in the left subfigure the word \"it\" refers to \"street\", while in the right subfigure the word \"it\" refers to \"animal\". Understanding the relationships between the words in such sentences has been challenging with traditional NLP approaches. Transformers use the self-attention mechanism to model the relationships between all words in a sentence, and assign weights to the other words in sentences based on their importance. In the left subfigure, the mechanism estimated that the ***query word*** \"it\" is most related to the word \"street\", but the word \"it\" is also somewhat related to the words \"The\" and \"animal. These words are referred to as ***key words*** for the query word \"it\".The intensity of the lines connecting the words as well as the intensity of the blue color signifies the attention weights or scores. The wider and bluer the lines, the higher the attention scores between two words are. \n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/attn_1.png' width=600px/>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specifically, Transformer Network compares each word to every other word in the sentence, to calculate an attention score. This is shown in the next figure, where for example, the word \"caves\" has the highest ***attention scores*** for the words \"glacier\" and \"formed\". The attention scores are calculated as the dot (inner) product of the input representations of every two words. That is, for each Query word $Q$ and Key word $K$, the attention score is $Q\\cdot K$. \n",
        "\n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/attn_2.png' width=250px/>\n",
        "\n"
      ],
      "metadata": {
        "id": "pR2M0cOujAnl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we mentioned in the previous lecture, Transformers employ word embeddings for representing the individual words in text sequences (where each text sequence can have one or several sentences). Recall that ***word embeddings*** convert each word into a feature vector, such that feature vectors of words that have similar semantic meaning have close spatial positions in the embeddings space. Therefore, the attention scores are dot products of the feature vectors (embeddings) for each pair of words in sentences. \n",
        "\n",
        "The obtained attention scores for each word are next first scaled (by dividing the values by $\\sqrt d$) and afterward are normalized to be in the [0,1] range (by applying a softmax function). That is, the attention scores are calculated as $a_{ij}=softmax(\\frac{Q_i\\cdot K_j}{\\sqrt d})$, where $d$ is the dimensionality of the feature vectors in the embedding space. Scaling the values is helpful to improve the flow of the gradients during training. The resulting scaled and normalized attention scores are then multiplied with the initial representation of the words, which in the self-attention module is referred to as ***value*** or $V$. \n",
        "\n",
        "This is shown in the next figure. The left subfigure shows the attention scores calculated as product of the input representations of the words $Q$ and $K$, which are afterwards multiplied with the input representations $V$ to obtain the output of the module. Note that for text classification, all three terms Query, Key, and Value are the input representation of the words in sentences. However, the original Transformer was developed for machine translation, where the words in the target language are queries, and the words in the source language are pairs of keys and values. This terminology is also related to search engines, which compare queries to keys, and determine values. Self-attention works in a similar way, where each query word is matched to other key words, and a weighted value is returned. \n",
        "\n",
        "The right subfigure below shows how self-attention is implemented in Transformer Networks. Namely, `Matmul` stands for a matrix multiplication layer which calculates the dot product $Q\\cdot K$, which is afterwards scaled by $\\sqrt d$, then there is an optional masking layer, and afterward the final attention scores are obtained by passing it through a `Softmax` layer to obtain $softmax(\\frac{Q_i\\cdot K_j}{\\sqrt d})$. Finally, the attention scores are multiplied with $V$ via another matrix multiplication layer `Matmul` to calculate the output of the self-attention module. \n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/attn_3.png' width=400px/>"
      ],
      "metadata": {
        "id": "UiIXXTnSwvXs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In conclusion, self-attention is applied to determine the meaning of the words in a sentence based on the context. That is, Transformers use the attention scores to modify the input vector representations for each word and generate a new representation based on the context of the sentence. During the training of the network, the representations of the words are updated and projected into a new embeddings space that takes the context into account."
      ],
      "metadata": {
        "id": "It3c8ihEcKXV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section3'></a>\n",
        "\n",
        "# 19.3 Multi-Head Attention"
      ],
      "metadata": {
        "id": "eYLfWGw6pN-Y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Transformer Networks include multiple self-attention modules in their architecture. Each self-attention module is called ***attention head***, and the aggregation of the outputs of multiple attention heads is called ***multi-head attention***. For instance, the original Transformer model had 8 attention heads, while the GPT-3 language model has 96 attention heads. \n",
        "\n",
        "The multi-head attention module is shown in the next figure, where the inputs are first passed through a linear layer (i.e., dense or fully-connected layer), next they are fed to the multiple attention heads, and the outputs of all attention heads are concatenated, and passed through one more linear (dense) layer. \n",
        "\n",
        "A logical question one can ask is why are multiple attention heads needed? The reason is that multiple attention modules can learn different relationships between the words in sentences. Each module can extract context independently from the other modules, which allows to capture less obvious context and enhance the learning capabilities of the model. Also, the computations of each attention head can be performed in parallel on different workers, which allows for accelerating the training and scaling up the models.\n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/multihead_1.png' width=500px/>"
      ],
      "metadata": {
        "id": "-jIbvAu3qj02"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section4'></a>\n",
        "\n",
        "# 19.4 Encoder Block"
      ],
      "metadata": {
        "id": "cQJzrp1Kqk1O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The ***Encoder Block*** in Transformer Networks is shown in the next figure. It processes the input embeddings of words and extracts representations in text data that can afterwards be used for different NLP tasks.\n",
        "\n",
        "The components in the Encoder Block are:\n",
        "- *Multi-head Attention layer*, which as explained, consists of multiple self-attention modules.\n",
        "- *Dropout layer*, is a regular dropout layer.\n",
        "- *Residual connections*, are skip connections in neural networks, where the input to a layer is added to the processed output of the layer. Residual connections were popularized in the ResNets models, as they were shown to stabilize the training phase of neural networks, and mitigate the problems of *vanishing and exploding gradients* (i.e., they refer to cases when the gradients become too small or too large during training). The `Add` term in the layer refers to the residual connection, which adds the input embeddings to the output of the Dropout layer.\n",
        "- *Layer Normalization*, is an operation that is similar to the batch normalization in CNN, but instead, it normalizes each sequence of words independently from the other sequences of words in the batch, and scales the data to have 0 mean and 1 standard deviation. This type of normalization is more adequate for text data. And, as we learned in the previous lectures, such normalization layers improve the flow of gradients during training. The `Norm` term in the figure refers to the Layer Normalization operation.\n",
        "- *Feed Forward network*, consists of 2 fully-connected layers that extract useful data representations.\n",
        "- The Encoder Block also contains one more *Dropout layer*, and another *Add & Norm* layer that forms a residual connection for the input to the Feed Forward network and applies a layer normalization operation.\n",
        "\n",
        "Larger Transformer networks typically include several encoder blocks in a sequence. For instance, in the original paper the authors used 6 encoder blocks.\n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/enc_1.png' width=250px/>"
      ],
      "metadata": {
        "id": "URSQk7VvFH8m"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The implementation of the Encoder Block in Keras and TensorFlow is shown in the cell following the imported libraries. \n",
        "\n",
        "The Encode Block is implemented as a layer which is a subclass of the `layers.Layer` class. The `__init__()` constructor method lists the definitions of the layers in the Encoder, and the method `call` provides the forward pass with the flow of information through the layers. \n",
        "- *Multi-head attention* layer is implemented in Keras, and it can be directly imported. The arguments in the layer are: `num_heads` the number of attention heads, and `key_dim` is the dimension of the embeddings of the input tokens.\n",
        "- *Dropout* and *Normalization* layers are also directly imported, with arguments `rate` for the dropout rate, and `epsilon` is a small float added to the standard deviation to avoid division by 0.\n",
        "- *Feed forward network* includes 2 dense layers, with the number of neurons set to `ff_dim` and `embed_dim`, respectively.\n",
        "- Also note the residual connections that are implemented in the layer normalization, e.g., the inputs are added to the output of the multi-head attention. "
      ],
      "metadata": {
        "id": "7PT8eNvBMUwo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers"
      ],
      "metadata": {
        "id": "31UvwJzcMVxg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class TransformerEncoder(layers.Layer):\n",
        "    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
        "        super().__init__()\n",
        "        self.multi_head_attention = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.feed_forward_net = keras.Sequential([layers.Dense(ff_dim, activation=\"relu\"), layers.Dense(embed_dim),])\n",
        "        self.layer_normalization1 = layers.LayerNormalization(epsilon=1e-6)\n",
        "        self.layer_normalization2 = layers.LayerNormalization(epsilon=1e-6)\n",
        "        self.dropout1 = layers.Dropout(rate)\n",
        "        self.dropout2 = layers.Dropout(rate)\n",
        "\n",
        "    def call(self, inputs, training):\n",
        "        multi_head_att_output = self.multi_head_attention(inputs, inputs)\n",
        "        multi_head_att_dropout = self.dropout1(multi_head_att_output, training=training)\n",
        "        add_norm_output_1 = self.layer_normalization1(inputs + multi_head_att_dropout)\n",
        "        feed_forward_output = self.feed_forward_net(add_norm_output_1)\n",
        "        feed_forward_dropout = self.dropout2(feed_forward_output, training=training)\n",
        "        add_norm_output_2 = self.layer_normalization2(add_norm_output_1 + feed_forward_dropout)\n",
        "        return add_norm_output_2"
      ],
      "metadata": {
        "id": "wk1rur6XMgPd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section5'></a>\n",
        "\n",
        "# 19.5 Positional Encoding"
      ],
      "metadata": {
        "id": "CaQtYLw6u2WW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We mentioned that Transformers use word embeddings as inputs, however, the embeddings alone don't provide information about the order of words in sentences. Understandably, the order of the words in a sentence is important, and different order of the words can convey a different meaning. To provide such information, Transformer Network introduces ***positional encoding*** for each word that is added to the input embedding, as shown in the next figure.  \n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/positional_encoding_1.png' width=250px/>"
      ],
      "metadata": {
        "id": "idmC_UGSu2oU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are different ways in which positional encoding can be implemented. In the original Transformer paper, the positional encoding is a vector that has the same size as the word embedding vector, and the authors used sine and cosine functions to create position vectors, which are afterwards scaled to be in the range from -1 to 1. Using such positional encoding, each encoding vector corresponds to a unique position in a sequence of words. This type is called *sinusoidal positional encoding*. \n",
        "\n",
        "The following cell implements the addition of positional encoding to word embeddings in Keras. In this case, we will not use the approach for obtaining positional encodings based on sine and cosine functions, but instead we will use a simpler approach and learn the positional encodings in the same way the word embeddings are learned. This type of positional encoding is referred to as *learned positional embeddings*. Therefore, for both token and positional embeddings we will use the `Embedding` layer in Keras which we introduced in the previous lecture. The arguments in the `Embedding` layer are the input dimension `input_dim` and the dimension of the embedding vectors `output_dim`. For the token embeddings layer, the input dimension is the size of the vocabulary, whereas for the positional embeddings layer the input dimension is the length of the text sequences. \n",
        "\n",
        "\n",
        "In the `call` method, first the length of the text sequences is assigned to `maxlen`. The function `tf.range` is similar to NumPy's `linspace` and creates numbers in the range from `start` to `limit` with a step `delta`. Next, the two separate `Embedding` layers are called, and returned is the sum of the token and positional embeddings."
      ],
      "metadata": {
        "id": "kW2zngtqyrDG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class TokenAndPositionEmbedding(layers.Layer):\n",
        "    def __init__(self, maxlen, vocab_size, embed_dim):\n",
        "        super().__init__()\n",
        "        self.token_embeddings = layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
        "        self.positional_embeddings = layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
        "\n",
        "    def call(self, inputs):\n",
        "        maxlen = tf.shape(inputs)[-1]\n",
        "        positions = tf.range(start=0, limit=maxlen, delta=1)\n",
        "        position_embeddings = self.positional_embeddings(positions)\n",
        "        input_embeddings = self.token_embeddings(inputs)\n",
        "        return input_embeddings + position_embeddings"
      ],
      "metadata": {
        "id": "sE9FKBCwu3CH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section6'></a>\n",
        "\n",
        "# 19.6 Using a Transformer Model for Classification"
      ],
      "metadata": {
        "id": "cu4fWFqH7Xt_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model Definition\n",
        "\n",
        "We will now employ the layers that we defined above, to create a Transformer model for text classification. \n",
        "\n",
        "It is a simple model that consists of the following parts:\n",
        "- ***Encoder***, which includes an `Input` layer that defines the maximum length of input sequences, `TokenAndPositionEmbedding` layer, and the `TransformerEncoder` layer.\n",
        "- ***Classifier***, which consists of a `GlobalAveragePooling1D` layer, and two `Dropout` and `Dense` layers. The Encoder block outputs a feature representation vector for each word in input text sequences. Global Average Pooling calculates the average value for each word, and it passes those values to the dense layers to classify the text sequences. "
      ],
      "metadata": {
        "id": "CTj6ItI1dOiu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "maxlen = 200  # Maximum length of input sequences is 200 words\n",
        "embed_dim = 32  # Embedding size for each token\n",
        "num_heads = 2  # Number of attention heads\n",
        "ff_dim = 32  # Dense layer size in the feed forward network inside transformer\n",
        "vocab_size = 20000  # The size of the vocabulary is 20k words\n",
        "\n",
        "# encoder\n",
        "inputs = layers.Input(shape=(maxlen,))\n",
        "embedding_layer = TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim)(inputs)\n",
        "x = TransformerEncoder(embed_dim, num_heads, ff_dim)(embedding_layer)\n",
        "\n",
        "# classifier\n",
        "x = layers.GlobalAveragePooling1D()(x)\n",
        "x = layers.Dropout(0.1)(x)\n",
        "x = layers.Dense(20, activation=\"relu\")(x)\n",
        "x = layers.Dropout(0.1)(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)"
      ],
      "metadata": {
        "id": "oCS2ZvAU7jC0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The summary of the model is shown below."
      ],
      "metadata": {
        "id": "SVGtqsUu-QpA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sya-beL_-Ljd",
        "outputId": "927de9aa-672e-4a4d-83c7-7f82804d8dab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 200)]             0         \n",
            "                                                                 \n",
            " token_and_position_embeddin  (None, 200, 32)          646400    \n",
            " g (TokenAndPositionEmbeddin                                     \n",
            " g)                                                              \n",
            "                                                                 \n",
            " transformer_encoder (Transf  (None, 200, 32)          10656     \n",
            " ormerEncoder)                                                   \n",
            "                                                                 \n",
            " global_average_pooling1d (G  (None, 32)               0         \n",
            " lobalAveragePooling1D)                                          \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 32)                0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 20)                660       \n",
            "                                                                 \n",
            " dropout_3 (Dropout)         (None, 20)                0         \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 1)                 21        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 657,737\n",
            "Trainable params: 657,737\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Loading the Dataset \n",
        "\n",
        "Let's apply the model for sentiment analysis of the movie reviews in the IMDB database. The data is loaded from the Keras datasets, and it contains 25,000 training sequences and 25,000 validation sequences. "
      ],
      "metadata": {
        "id": "AlDGBEt8-e4J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "(x_train, y_train), (x_val, y_val) = keras.datasets.imdb.load_data(num_words=vocab_size)\n",
        "print(len(x_train), \"Training sequences\")\n",
        "print(len(x_val), \"Validation sequences\")\n",
        "x_train = keras.preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "x_val = keras.preprocessing.sequence.pad_sequences(x_val, maxlen=maxlen)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNe755zF-h_f",
        "outputId": "fdbf1a27-46be-49ab-dbf7-de5d1ef5c1e7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17464789/17464789 [==============================] - 0s 0us/step\n",
            "25000 Training sequences\n",
            "25000 Validation sequences\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model Training "
      ],
      "metadata": {
        "id": "GEXdQPdg-X4m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.compile(optimizer=\"adam\", loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n",
        "\n",
        "model.fit(x_train, y_train, batch_size=32, epochs=2, validation_data=(x_val, y_val))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1mmvdi3a-YQ_",
        "outputId": "13e64460-5cc7-4874-a652-dbee90b0feef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/2\n",
            "782/782 [==============================] - 13s 11ms/step - loss: 0.3946 - accuracy: 0.8092 - val_loss: 0.3095 - val_accuracy: 0.8680\n",
            "Epoch 2/2\n",
            "782/782 [==============================] - 9s 11ms/step - loss: 0.2045 - accuracy: 0.9213 - val_loss: 0.3259 - val_accuracy: 0.8744\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7fd2552d61d0>"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section7'></a>\n",
        "\n",
        "# 19.7 Fine-tuning a Pretrained BERT Model"
      ],
      "metadata": {
        "id": "AqW48ZemAIbO"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qxOWDkU-Z3jj"
      },
      "source": [
        "***BERT*** (Bidirectional Encoder Representations from Transformers) is a Transformer Network, and a language model that can be used for variety of NLP tasks such as question answering, text classification, machine translation, etc. \n",
        "\n",
        "In this section we will use a pretrained version of BERT and fine-tuned it for classification of news articles in the AG database (that we used in the previous lecture). \n",
        "\n",
        "TensorFlow Hub is a repository of pretrained machine learning models, and it offers several versions of [BERT](https://tfhub.dev/google/collections/bert/1) such as: [Small BERT](https://tfhub.dev/google/collections/bert/1), [Albert](https://tfhub.dev/google/collections/albert/1), and [BERT Expert](https://tfhub.dev/google/collections/experts/bert/1). The different versions of BERT are optimized for different use cases. In our case, we will use [SmallBERT](https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3). \n",
        "\n",
        "To use this model we will need to install the TensorFlow Text library for text processing."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow_text\n",
        "import tensorflow_text as text"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BgRpU8iyENG6",
        "outputId": "60895598-84a8-46eb-ae75-a7a6079bb635"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: tensorflow_text in /usr/local/lib/python3.7/dist-packages (2.10.0)\n",
            "Requirement already satisfied: tensorflow-hub>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_text) (0.12.0)\n",
            "Requirement already satisfied: tensorflow<2.11,>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_text) (2.10.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (3.1.0)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.21.6)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (0.27.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (2.0.1)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (3.3.0)\n",
            "Requirement already satisfied: flatbuffers>=2.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (22.9.24)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.49.1)\n",
            "Requirement already satisfied: tensorboard<2.11,>=2.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (2.10.1)\n",
            "Requirement already satisfied: tensorflow-estimator<2.11,>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (2.10.0)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (14.0.6)\n",
            "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (0.4.0)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.3.0)\n",
            "Requirement already satisfied: keras<2.11,>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (2.10.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.15.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.1.2)\n",
            "Requirement already satisfied: protobuf<3.20,>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (3.17.3)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (21.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (57.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (4.1.1)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.6.3)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (0.2.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.11,>=2.10.0->tensorflow_text) (1.14.1)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow<2.11,>=2.10.0->tensorflow_text) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.5.2)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.35.0)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.0.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.8.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (2.23.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (0.6.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (0.4.6)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (3.4.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (4.9)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (4.2.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (4.13.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (3.9.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (0.4.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (2022.9.24)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (1.24.3)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.11,>=2.10->tensorflow<2.11,>=2.10.0->tensorflow_text) (3.2.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->tensorflow<2.11,>=2.10.0->tensorflow_text) (3.0.9)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The BERT model in TensorFlow Hub has a corresponding text preprocessing model for converting  texts into tokens. "
      ],
      "metadata": {
        "id": "sMEEA9-yE-8E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_hub as hub\n",
        "\n",
        "bert_handle = 'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/2'\n",
        "preprocessing_model = 'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'"
      ],
      "metadata": {
        "id": "drbAfm7HFC59"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYjYF5jDEEdk"
      },
      "source": [
        "The output of the `preprocessing model` has 3 elements:  \n",
        "- `input_word_ids`: token ids of the input sequences.\n",
        "- `input_mask`: has value 1 for all input tokens before padding, and value 0 for the padding tokens.\n",
        "- `input_type_ids`: has different values for segments in text; e.g., if there are 3 sentences in the input text, the tokens in the same sentences will have the same index."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGHlY82i_Gde"
      },
      "source": [
        "Let's wrap `preprocessing_model` into a `hub.KerasLayer` and test it on a sample sentence. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-8iJ0dhE_WHh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f71781f-bafe-472b-a89e-f0f12125f4a5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11.\n"
          ]
        }
      ],
      "source": [
        "preprocess_layer = hub.KerasLayer(preprocessing_model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NpLEBPAK_sEn",
        "outputId": "083d7fc9-d6cc-4e98-9833-459ce6970d43"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Keys: dict_keys(['input_type_ids', 'input_mask', 'input_word_ids'])\n",
            "Shape: (1, 128)\n",
            "Word Ids: tf.Tensor([  101  6627 11256  1024  1996  6627  5016  6207  2003  2551], shape=(10,), dtype=int32)\n",
            "Input Mask: tf.Tensor([1 1 1 1 1 1 1 1 1 1], shape=(10,), dtype=int32)\n",
            "Type Ids: tf.Tensor([0 0 0 0 0 0 0 0 0 0], shape=(10,), dtype=int32)\n"
          ]
        }
      ],
      "source": [
        "sample_news = ['Tech rumors: The tech giant Apple is working on a self driving car']\n",
        "\n",
        "preprocessed_news = preprocess_layer(sample_news)\n",
        "\n",
        "print('Keys:', preprocessed_news.keys())\n",
        "# length of the input sequence\n",
        "print('Shape:', preprocessed_news[\"input_word_ids\"].shape)\n",
        "print('Word Ids:', preprocessed_news[\"input_word_ids\"][0,:10])\n",
        "print('Input Mask:', preprocessed_news[\"input_mask\"][0, :10])\n",
        "print('Type Ids:', preprocessed_news[\"input_type_ids\"][0, :10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BwdYMcP3vb4K"
      },
      "source": [
        "### Loading the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSyKsBxnXEY3"
      },
      "source": [
        "The news articles in the AG dataset are classified into 4 categories: World, Sports, Business, and Sci/Tech."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6JIKIZDRQwVV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 148,
          "referenced_widgets": [
            "0f098a5cb3ec4cb5ad54a183f631a317",
            "6a1d484e3538431590b1cd7530b69963",
            "2eb6aa377d8c4960b40a9f97e03ea303",
            "99264c7c98a64f16875dadd82e1a9c42",
            "accddde4ab8e4036a5af11b5467e65a3",
            "de186becd1ff4664a6a71068effaaa13",
            "b2d410f8f6f34e61ad66f6365b95f4af",
            "08311b7e4bad4ed8b7be6563c44448f4",
            "3c0ff08964ec4c0684c49082a6bda252",
            "c51af815fab243318bd19370424ba8bf",
            "952f8d37d2bb470092a8477f95163a37",
            "2a3703e6d5c9469dbec3bf7e771acf80",
            "4b54d440726140d0afe7e83671eabb57",
            "c9ee45c2c34242f7afb6f3b663a30af4",
            "fea5aff80c7a4a82b1687cfa2340240e",
            "b86cdda42c174f8882417d8ff4ca443b",
            "ada5dd2e2d6344b698508d5145bfb727",
            "847c7906c8dd4e56b6517bec0346e189",
            "00f23294543c4be7b4a53ba05b9fd605",
            "08b4ae0dc59b48d08ccd235fa0230591",
            "d3d2e3c3e9d8462dab569fc0eac18750",
            "2d79d482110e4285a496b18a288aa00e",
            "c46f414771754547912aea4d972595e0",
            "200dfaa54e5543c59849cf3b76f8730f",
            "258d6afa25f145b5b35237cac9830140",
            "783b0a49072e4445b7573922679fa055",
            "a2c73100cde445f09d3b5640b4353246",
            "c17d5683b71a413da79f17782370d455",
            "26f5c1d8f6144c9aae467577012bb6cb",
            "97b03fe8bc114421b3b1bd54a8890ae0",
            "6b21b9f0c7c6481faba90cf1f6ff725f",
            "0a387adc4ae8466db5732fdf24cdd68a",
            "5d0054f2907c4fc8adbbefc9e2d68288",
            "f0997556800b4e51bcf78fcdb5e8295e",
            "909cefafb5a64bd499930d612600d5d0",
            "034212a4e741495597bf1e5aca871a16",
            "429399ddb6c44c19a57c1661194c2791",
            "acef820e431b4333b48ed60c78cc9311",
            "1f6c71ba286849a087c3ebf698b8fa17",
            "726dc391be0f401bb0b602031cf59533",
            "543bb07fb68546428077e9b11eb56e50",
            "c08af6f4a64e45b29165b9ccddd35e9f",
            "05c333dcfbb54524b9c4979cf2dfedbf",
            "3ffd432dd7d24cfbb0cf4fd1629f41bb",
            "bd81abe140984e53acf8f44e8c147d22",
            "cd560736edae484c9a89d99a3f8550c8",
            "9a6c991e11f74e1394a47144dafccea6",
            "9ad549df37b342d49910f296a8f5ec1e",
            "e83ca730895944febe01e58c3f0f6f3d",
            "7eb6e90ce93040509d231891b07cb850",
            "30fd9ab51a5c4dbdbedf7236f25dd597",
            "dd2745d64f474f0e989f812f5de682bb",
            "81e0784d03244f0784401ce535dadf08",
            "cfcd5ebdbe90405f981ed6b8c82f00cd",
            "cd4496a4380e4370a3d70848c175aeb2",
            "b96ea36ae6c34268b3a00932329041af",
            "9005ed3d29a84439bee67da450250682",
            "e4c4c57e171947728d552acdc618cef3",
            "d0ab68e04b954ea4bfe983ce23e7b135",
            "14676964e80b401e9f73d7e46d1d2f3e",
            "d75d6bd5104b44aaa88b3e3bdc11d884",
            "8a0b7f8c73944a7d9abfcfd5525344fa",
            "acaaa7ad4ee4409f894dae6a32ae444e",
            "85dbc689123540e58a343905eae2aec0",
            "b2514562cade4be982537f0e3eaa06ae",
            "54eafeb7352341238077208abc56a753",
            "c3a6f01347494d0ab6744a9e878cabfa",
            "bd02e7658a0e46a1aecc8fc959ffc44c",
            "4590d63e2bfa4df5b99e54623ccbb4f6",
            "8cfde95e14fe4181a456ff150e8a09cf",
            "bbbbceb91a434da591df82e17696efdf",
            "8532a6042986455eb202e0dc027df34c",
            "08a74f470b7f43dcb37cd8ddb6f02a8d",
            "e1b40d82557c4362babea2699a9eac5c",
            "9f9b72704bfa4294aac194062c0b71e4",
            "3e6954eb93d24a91a35b14f5865492f7",
            "1e404b3f054547af875603e31a936f29",
            "c8d130cea21d47359afb5c7b84a74ef1",
            "67fd73d20c2b46e8b7967d42bc49129b",
            "a169a9583c24471f8b395aadc7a62b92",
            "78130d3bb3bc4c48b52f229113b77a62",
            "1958fdfb3407464cb4aeba87cd581466",
            "6fbe75168d5d4091919116209bc1be74",
            "a1efa58385ac4978b32468398c1b7c71",
            "1150c0c0c59947fcaf3ebc98ecee7ada",
            "a749bf6f121b499399887d6800af56ed",
            "55fcc48243474f5894ea2841668544bb",
            "002a2a020344451ab1e251b28788a06f"
          ]
        },
        "outputId": "c84f27df-c13f-491a-f202-42afd8de175e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1mDownloading and preparing dataset 11.24 MiB (download: 11.24 MiB, generated: 35.79 MiB, total: 47.03 MiB) to ~/tensorflow_datasets/ag_news_subset/1.0.0...\u001b[0m\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Dl Completed...: 0 url [00:00, ? url/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0f098a5cb3ec4cb5ad54a183f631a317"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Dl Size...: 0 MiB [00:00, ? MiB/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2a3703e6d5c9469dbec3bf7e771acf80"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Extraction completed...: 0 file [00:00, ? file/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c46f414771754547912aea4d972595e0"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating splits...:   0%|          | 0/2 [00:00<?, ? splits/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f0997556800b4e51bcf78fcdb5e8295e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train examples...:   0%|          | 0/120000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bd81abe140984e53acf8f44e8c147d22"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Shuffling ~/tensorflow_datasets/ag_news_subset/1.0.0.incompleteS3CHIE/ag_news_subset-train.tfrecord*...:   0%|…"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b96ea36ae6c34268b3a00932329041af"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating test examples...:   0%|          | 0/7600 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c3a6f01347494d0ab6744a9e878cabfa"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Shuffling ~/tensorflow_datasets/ag_news_subset/1.0.0.incompleteS3CHIE/ag_news_subset-test.tfrecord*...:   0%| …"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c8d130cea21d47359afb5c7b84a74ef1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1mDataset ag_news_subset downloaded and prepared to ~/tensorflow_datasets/ag_news_subset/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "\n",
        "(train_data, val_data), info = tfds.load('ag_news_subset:1.0.0', #version 1.0.0\n",
        "                                         split=('train', 'test'),\n",
        "                                         with_info=True, \n",
        "                                         as_supervised=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "03IBh4d1S1eJ",
        "outputId": "c2535bfb-c5dd-4436-d420-7933d9432f8e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Classes: ['World', 'Sports', 'Business', 'Sci/Tech']\n",
            "Number of training samples\" 120000\n",
            "Number of test samples\" 7600\n"
          ]
        }
      ],
      "source": [
        "# Dataset information\n",
        "class_names = info.features['label'].names\n",
        "print('Classes:', class_names)\n",
        "\n",
        "print('Number of training samples\"', info.splits['train'].num_examples)\n",
        "print('Number of test samples\"', info.splits['test'].num_examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3e-BTCUfmdr"
      },
      "outputs": [],
      "source": [
        "buffer_size = 1000\n",
        "batch_size = 32\n",
        "\n",
        "# prepare the data\n",
        "train_data = train_data.shuffle(buffer_size)\n",
        "train_data = train_data.batch(batch_size).prefetch(1)\n",
        "val_data = val_data.batch(batch_size).prefetch(1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1_z6U5WA9Pq"
      },
      "source": [
        "### Model Definition with BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wyPUdJ8aBQmZ"
      },
      "source": [
        "The model defined below includes an Input layer, a preprocessing layer to convert the text data into token embeddings, and a layer for the BERT model.\n",
        "\n",
        "Afterward, the output is passed through a classifier head, which includes two dense layers and dropout layers. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5sqJYq7wBCWO"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "import numpy as np\n",
        "\n",
        "# input layer\n",
        "input_text = layers.Input(shape=(), dtype=tf.string)\n",
        "\n",
        "# preprocesing model\n",
        "preprocessing_layer = hub.KerasLayer(preprocessing_model)(input_text)\n",
        "# Bert model, set trainable to True\n",
        "bert_encoder = hub.KerasLayer(bert_handle, trainable=True)(preprocessing_layer)\n",
        "\n",
        "# For fine-tuning use pooled output \n",
        "pooled_bert_output = bert_encoder['pooled_output']\n",
        "\n",
        "# clasifier\n",
        "x = layers.Dense(16, activation='relu')(pooled_bert_output)\n",
        "x = layers.Dropout(0.2)(x)\n",
        "final_output = keras.layers.Dense(4, activation='softmax')(x)\n",
        "\n",
        "\n",
        "# Combine input and output\n",
        "news_model = keras.Model(input_text, final_output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rMP5y7SwKJS9"
      },
      "source": [
        "### Model Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDYULiuLKPcD"
      },
      "source": [
        "Let's compile and train the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lh2sQNQ5JlWz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ac61b58-4de7-4600-9812-2fa28f913dd7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n",
            "3750/3750 [==============================] - 725s 192ms/step - loss: 0.3140 - accuracy: 0.8954 - val_loss: 0.2316 - val_accuracy: 0.9222\n",
            "Epoch 2/3\n",
            "3750/3750 [==============================] - 717s 191ms/step - loss: 0.2100 - accuracy: 0.9310 - val_loss: 0.2143 - val_accuracy: 0.9279\n",
            "Epoch 3/3\n",
            "3750/3750 [==============================] - 717s 191ms/step - loss: 0.1620 - accuracy: 0.9480 - val_loss: 0.2468 - val_accuracy: 0.9245\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7fa662301910>"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "# compile\n",
        "news_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5), \n",
        "                        loss='sparse_categorical_crossentropy', \n",
        "                        metrics=['accuracy'])\n",
        "# train \n",
        "news_model.fit(train_data, epochs=3, validation_data=val_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGX0X6PVOE8K"
      },
      "source": [
        "### Model Evaluation \n",
        "\n",
        "Finally, let's predict the class of two news articles."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vx4pDaAeOPJi",
        "outputId": "61b58aa2-ea92-40b0-9138-8b145109d345"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 395ms/step\n",
            "Predicted class: 3\n",
            "Predicted class name: Sci/Tech\n"
          ]
        }
      ],
      "source": [
        "sample_news_1 = ['Tesla, a self driving car company is also planning to make a humanoid robot. This humanoid robot appeared dancing in the latest Tesla AI day']\n",
        "\n",
        "predictions_1 = news_model.predict(np.array(sample_news_1))\n",
        "\n",
        "predicted_class_1 = np.argmax(predictions_1)\n",
        "\n",
        "print('Predicted class:', predicted_class_1)\n",
        "print('Predicted class name:', class_names[predicted_class_1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6RrCqt3PORiJ",
        "outputId": "367c830f-00f1-4b85-df74-f910fcd22a2c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 23ms/step\n",
            "Predicted class: 1\n",
            "Predicted class name: Sports\n"
          ]
        }
      ],
      "source": [
        "sample_news_2 = [\"In the last weeks, there has been many transfer suprises in footbal. Ronaldo went back to Old Trafford, \"\n",
        "                \"while Messi went to Paris Saint Germain to join his former colleague Neymar.\"\n",
        "                \"We can't wait to see these two clubs will perform in upcoming leagues\"]\n",
        "            \n",
        "predictions_2 = news_model.predict(np.array(sample_news_2))\n",
        "\n",
        "predicted_class_2 = np.argmax(predictions_2)\n",
        "\n",
        "print('Predicted class:', predicted_class_2)\n",
        "print('Predicted class name:', class_names[predicted_class_2])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section8'></a>\n",
        "\n",
        "# 19.8 Decoder Sub-network"
      ],
      "metadata": {
        "id": "J9KU9cmkAMwM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Transformer Network in the original paper was designed for machine translation. Differently from the text classification task where for an input text sentence the model predicts a class label, in machine translation for an input text sentence in a source language the model predicts the corresponding text sentence in a target language. Therefore, both the input and output of the model are text sequences. These type of models are called ***sequence-to-sequence models***, or oftentimes this term is abbreviated to ***seq2seq models***. Beside machine translation, other NLP tasks that employ seq2seq models include question answering, text summarization, dialog generation, and others. \n",
        "\n",
        "The architecture of Transformer Networks designed to handle seq2seq tasks consists of encoder and decoder sub-networks.\n",
        "- ***Encoder sub-network*** takes a source text sequence as an input, and extracts a useful representation of the text data.\n",
        "- ***Decoder sub-network*** takes a target text sequence as an input, as well as it receives the intermediate representation from the encoder sub-network. The decoder combines the information from the target sequence and the encoded source sequence, and learns to predict the next word (token) in the target sequence. \n",
        "\n",
        "During the evaluation step, the model does not have access to the target sequence. It is just fed with a source sequence, and the model tries to predict the next word in the target sequence. Afterward, the predicted target sequence is fed back to the decoder, and the next word is again predicted. This step is repeated until the decoder generates an end-of-sequence token. Note also that such models that generate one token at each time step and feed back the output back to the model are called ***autoregressive models***.\n",
        "\n",
        "This is shown in the next figure, where the French sequence \"Je suis etudiant\" is translated into \"I am a student\". The decoder outputs one word at each time step until the end-of-sequence is reached.\n",
        "\n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/transformer_decoding_2.gif' width=700px/>"
      ],
      "metadata": {
        "id": "14Ix6YwoxmyL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The architecture of the decoder is similar to the encoder and it is shown in the next figure. The upper part of the decoder is practically the same as the encoder, and it consists of a multi-head attention module with residual connections and layer normalization, followed by a feed-forward network with residual connections and layer normalization. The output of the encoder is passed to the multi-head attention module.\n",
        "\n",
        "The main difference from the encoder is the *masked multi-head attention* module in the lower part of the decoder. This module is inserted between the target sequence (i.e., the output sequence of the decoder) and the multi-head attention module. Masked multi-head attention module applies masking to the next words in the target sequence, so that the network does not have access to those words. That is, during training, if the model needs to predict the 4th word in a sentence, masks will be applied to all words after the 3rd word, so that the model has access only to the words 1, 2, and 3, in order to predict the 4th word. This step ensures that the model uses only the previous steps to predict the word in the next step in the target sequence. This type of mask is also referred to as *causal attention mask*.\n",
        "\n",
        "Finally, the output representations from the decoder are inputted to a linear (dense) layer and a softmax layer, that outputs the probability for the next word in the vocabulary learned from the training dataset.\n",
        "\n",
        "And also note the marks `Nx` in the figure. They indicate that the shown modules in the encoder and decoder are repeated multiple times in the network. In the original Transformer Network, the encoder sub-network has 6 blocks of multi-head attention and feed forward modules, and similarly the decoder sub-network has 6 blocks of masked multi-head attention, multi-head attention, and feed forward modules. Introducing multiple modules in the sub-networks increases the learning ability as it allows the model to learn more abstract representations. \n",
        "\n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/transformer.png' width=450px/>"
      ],
      "metadata": {
        "id": "zDTIt98p6Pzm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that Recurrent Neural Networks are also a type of seq2seq models. Transformer Networks have several advantages over RNN, due to the ability to inspect entire text sequences at once, capture context in long sequences, are parallelizable, and are more powerful in general. Conversely, RNN have access only to the next tokens in a sequence (have difficulty finding correlations in long sequences because the information needs to pass through many processing steps), can not perform parallel computations (are slow to train), and the gradients can become unstable. "
      ],
      "metadata": {
        "id": "14HTvHQyFCqr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='section9'></a>\n",
        "\n",
        "# 19.9 Vision Transformers"
      ],
      "metadata": {
        "id": "b_PWJPleANCx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "After the initial success of Transformer Networks in NLP, recently they have been adapted for computer vision tasks as well. The initial Transformer model for vision tasks proposed in 2021 was called ***Vision Transformer (ViT)***. \n",
        "\n",
        "The architecture of ViT is very similar to the Transformers used in NLP. However, Transformer Networks were designed for working with sequential data, while images are spatial data types. To consider each pixel in an image as a sequential token would be impractical and too time-consuming. Therefore, ViT splits images into a set of smaller image patches (16x16 pixels), and it uses the sequence of image patches as inputs to the model. Each image patch was first flattened to one-dimensional vector, and those vectors were afterward passed through a dense layer to learn lower-dimensional embeddings for each patch. Positional embeddings were added, and the sequences were fed to a standard transformer encoder. The encoder block in ViT is identical to the encoder in the original Transformer Network. The steps are depicted in the figure below. \n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/vision_transformer.gif' width=700px/>"
      ],
      "metadata": {
        "id": "vj37AnHbLuWa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The authors trained 3 versions of ViT, called Base (12 blocks, 768 embeddings dimension, 86M parameters), Large (24 blocks, 1,024 embeddings dimension, 307M parameters), and Huge (32 blocks, 1,280 embeddings dimension, 632M parameters).\n",
        "\n",
        "Various other versions of vision transformers were introduced recently, which include MaxViT (Multi-axis ViT), Swin (Shifted Window ViT), DeiT (Data-efficient image Transformer), T2T-ViT (Token-to-token ViT), and others. These models achieved higher accuracy on many vision tasks in comparison to Convolutional Neural Networks. The following figure shows the current accuracy on ImageNet. \n",
        "\n",
        "<img src='https://raw.githubusercontent.com/avakanski/Fall-2022-Python-Programming-for-Data-Science/main/Lectures/Theme%203%20-%20Model%20Engineering%20Pipelines/Lecture%2019%20-%20Transformer%20Networks/images/imagenet_accuracy.png' width=500px/>\n"
      ],
      "metadata": {
        "id": "Msq8PaT_O-hh"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vweobvFVe4RB"
      },
      "source": [
        "<a name='section10'></a>\n",
        "\n",
        "# References\n",
        "\n",
        "1. The Illustrated Transformer, Jay Alammar, available at: [https://jalammar.github.io/illustrated-transformer/](https://jalammar.github.io/illustrated-transformer/).\n",
        "2. Keras Examples, Text classification with Transformer, available at: [https://keras.io/examples/nlp/text_classification_with_transformer/](https://keras.io/examples/nlp/text_classification_with_transformer/).\n",
        "3. Using Pretrained BERT for Text Classification, Jean de Dieu Nyandwi, available at: [https://github.com/Nyandwi/machine_learning_complete/blob/main/9_nlp_with_tensorflow/5_using_pretrained_bert_for_text_classification.ipynb](https://github.com/Nyandwi/machine_learning_complete/blob/main/9_nlp_with_tensorflow/5_using_pretrained_bert_for_text_classification.ipynb).\n",
        "4. Deep Learning with Python, Francois Chollet, Second Edition, Manning Publications, 2021.\n",
        "5. TensorFlow Tutorials, Neural Machine Translation with a Transformer and Keras, available at [https://www.tensorflow.org/text/tutorials/transformer](https://www.tensorflow.org/text/tutorials/transformer).\n",
        "6. How the Vision Transformer (ViT) Works in 10 Minutes: An Image is Worth 16x16 Words, Nikolas Adaloglou, available at [https://theaisummer.com/vision-transformer/](https://theaisummer.com/vision-transformer/).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDbzcXmWe4RB"
      },
      "source": [
        "[BACK TO TOP](#section0)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0f098a5cb3ec4cb5ad54a183f631a317": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6a1d484e3538431590b1cd7530b69963",
              "IPY_MODEL_2eb6aa377d8c4960b40a9f97e03ea303",
              "IPY_MODEL_99264c7c98a64f16875dadd82e1a9c42"
            ],
            "layout": "IPY_MODEL_accddde4ab8e4036a5af11b5467e65a3"
          }
        },
        "6a1d484e3538431590b1cd7530b69963": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_de186becd1ff4664a6a71068effaaa13",
            "placeholder": "​",
            "style": "IPY_MODEL_b2d410f8f6f34e61ad66f6365b95f4af",
            "value": "Dl Completed...: 100%"
          }
        },
        "2eb6aa377d8c4960b40a9f97e03ea303": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_08311b7e4bad4ed8b7be6563c44448f4",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3c0ff08964ec4c0684c49082a6bda252",
            "value": 1
          }
        },
        "99264c7c98a64f16875dadd82e1a9c42": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c51af815fab243318bd19370424ba8bf",
            "placeholder": "​",
            "style": "IPY_MODEL_952f8d37d2bb470092a8477f95163a37",
            "value": " 1/1 [00:03&lt;00:00,  2.74s/ url]"
          }
        },
        "accddde4ab8e4036a5af11b5467e65a3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de186becd1ff4664a6a71068effaaa13": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b2d410f8f6f34e61ad66f6365b95f4af": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "08311b7e4bad4ed8b7be6563c44448f4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "3c0ff08964ec4c0684c49082a6bda252": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c51af815fab243318bd19370424ba8bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "952f8d37d2bb470092a8477f95163a37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2a3703e6d5c9469dbec3bf7e771acf80": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4b54d440726140d0afe7e83671eabb57",
              "IPY_MODEL_c9ee45c2c34242f7afb6f3b663a30af4",
              "IPY_MODEL_fea5aff80c7a4a82b1687cfa2340240e"
            ],
            "layout": "IPY_MODEL_b86cdda42c174f8882417d8ff4ca443b"
          }
        },
        "4b54d440726140d0afe7e83671eabb57": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ada5dd2e2d6344b698508d5145bfb727",
            "placeholder": "​",
            "style": "IPY_MODEL_847c7906c8dd4e56b6517bec0346e189",
            "value": "Dl Size...: 100%"
          }
        },
        "c9ee45c2c34242f7afb6f3b663a30af4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_00f23294543c4be7b4a53ba05b9fd605",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_08b4ae0dc59b48d08ccd235fa0230591",
            "value": 1
          }
        },
        "fea5aff80c7a4a82b1687cfa2340240e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d3d2e3c3e9d8462dab569fc0eac18750",
            "placeholder": "​",
            "style": "IPY_MODEL_2d79d482110e4285a496b18a288aa00e",
            "value": " 11/11 [00:03&lt;00:00,  1.87 MiB/s]"
          }
        },
        "b86cdda42c174f8882417d8ff4ca443b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ada5dd2e2d6344b698508d5145bfb727": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "847c7906c8dd4e56b6517bec0346e189": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "00f23294543c4be7b4a53ba05b9fd605": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "08b4ae0dc59b48d08ccd235fa0230591": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d3d2e3c3e9d8462dab569fc0eac18750": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d79d482110e4285a496b18a288aa00e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c46f414771754547912aea4d972595e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_200dfaa54e5543c59849cf3b76f8730f",
              "IPY_MODEL_258d6afa25f145b5b35237cac9830140",
              "IPY_MODEL_783b0a49072e4445b7573922679fa055"
            ],
            "layout": "IPY_MODEL_a2c73100cde445f09d3b5640b4353246"
          }
        },
        "200dfaa54e5543c59849cf3b76f8730f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c17d5683b71a413da79f17782370d455",
            "placeholder": "​",
            "style": "IPY_MODEL_26f5c1d8f6144c9aae467577012bb6cb",
            "value": "Extraction completed...: 100%"
          }
        },
        "258d6afa25f145b5b35237cac9830140": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_97b03fe8bc114421b3b1bd54a8890ae0",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6b21b9f0c7c6481faba90cf1f6ff725f",
            "value": 1
          }
        },
        "783b0a49072e4445b7573922679fa055": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0a387adc4ae8466db5732fdf24cdd68a",
            "placeholder": "​",
            "style": "IPY_MODEL_5d0054f2907c4fc8adbbefc9e2d68288",
            "value": " 1/1 [00:03&lt;00:00,  3.00s/ file]"
          }
        },
        "a2c73100cde445f09d3b5640b4353246": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c17d5683b71a413da79f17782370d455": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "26f5c1d8f6144c9aae467577012bb6cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "97b03fe8bc114421b3b1bd54a8890ae0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "6b21b9f0c7c6481faba90cf1f6ff725f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0a387adc4ae8466db5732fdf24cdd68a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5d0054f2907c4fc8adbbefc9e2d68288": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f0997556800b4e51bcf78fcdb5e8295e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_909cefafb5a64bd499930d612600d5d0",
              "IPY_MODEL_034212a4e741495597bf1e5aca871a16",
              "IPY_MODEL_429399ddb6c44c19a57c1661194c2791"
            ],
            "layout": "IPY_MODEL_acef820e431b4333b48ed60c78cc9311"
          }
        },
        "909cefafb5a64bd499930d612600d5d0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1f6c71ba286849a087c3ebf698b8fa17",
            "placeholder": "​",
            "style": "IPY_MODEL_726dc391be0f401bb0b602031cf59533",
            "value": "Generating splits...: 100%"
          }
        },
        "034212a4e741495597bf1e5aca871a16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_543bb07fb68546428077e9b11eb56e50",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c08af6f4a64e45b29165b9ccddd35e9f",
            "value": 2
          }
        },
        "429399ddb6c44c19a57c1661194c2791": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_05c333dcfbb54524b9c4979cf2dfedbf",
            "placeholder": "​",
            "style": "IPY_MODEL_3ffd432dd7d24cfbb0cf4fd1629f41bb",
            "value": " 2/2 [00:16&lt;00:00,  6.99s/ splits]"
          }
        },
        "acef820e431b4333b48ed60c78cc9311": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "1f6c71ba286849a087c3ebf698b8fa17": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "726dc391be0f401bb0b602031cf59533": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "543bb07fb68546428077e9b11eb56e50": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c08af6f4a64e45b29165b9ccddd35e9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "05c333dcfbb54524b9c4979cf2dfedbf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3ffd432dd7d24cfbb0cf4fd1629f41bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bd81abe140984e53acf8f44e8c147d22": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cd560736edae484c9a89d99a3f8550c8",
              "IPY_MODEL_9a6c991e11f74e1394a47144dafccea6",
              "IPY_MODEL_9ad549df37b342d49910f296a8f5ec1e"
            ],
            "layout": "IPY_MODEL_e83ca730895944febe01e58c3f0f6f3d"
          }
        },
        "cd560736edae484c9a89d99a3f8550c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7eb6e90ce93040509d231891b07cb850",
            "placeholder": "​",
            "style": "IPY_MODEL_30fd9ab51a5c4dbdbedf7236f25dd597",
            "value": "Generating train examples...: 100%"
          }
        },
        "9a6c991e11f74e1394a47144dafccea6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dd2745d64f474f0e989f812f5de682bb",
            "max": 120000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_81e0784d03244f0784401ce535dadf08",
            "value": 120000
          }
        },
        "9ad549df37b342d49910f296a8f5ec1e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cfcd5ebdbe90405f981ed6b8c82f00cd",
            "placeholder": "​",
            "style": "IPY_MODEL_cd4496a4380e4370a3d70848c175aeb2",
            "value": " 119636/120000 [00:15&lt;00:00, 8968.09 examples/s]"
          }
        },
        "e83ca730895944febe01e58c3f0f6f3d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "7eb6e90ce93040509d231891b07cb850": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30fd9ab51a5c4dbdbedf7236f25dd597": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "dd2745d64f474f0e989f812f5de682bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81e0784d03244f0784401ce535dadf08": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cfcd5ebdbe90405f981ed6b8c82f00cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd4496a4380e4370a3d70848c175aeb2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b96ea36ae6c34268b3a00932329041af": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9005ed3d29a84439bee67da450250682",
              "IPY_MODEL_e4c4c57e171947728d552acdc618cef3",
              "IPY_MODEL_d0ab68e04b954ea4bfe983ce23e7b135"
            ],
            "layout": "IPY_MODEL_14676964e80b401e9f73d7e46d1d2f3e"
          }
        },
        "9005ed3d29a84439bee67da450250682": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d75d6bd5104b44aaa88b3e3bdc11d884",
            "placeholder": "​",
            "style": "IPY_MODEL_8a0b7f8c73944a7d9abfcfd5525344fa",
            "value": "Shuffling ~/tensorflow_datasets/ag_news_subset/1.0.0.incompleteS3CHIE/ag_news_subset-train.tfrecord*...:  70%"
          }
        },
        "e4c4c57e171947728d552acdc618cef3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_acaaa7ad4ee4409f894dae6a32ae444e",
            "max": 120000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_85dbc689123540e58a343905eae2aec0",
            "value": 120000
          }
        },
        "d0ab68e04b954ea4bfe983ce23e7b135": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b2514562cade4be982537f0e3eaa06ae",
            "placeholder": "​",
            "style": "IPY_MODEL_54eafeb7352341238077208abc56a753",
            "value": " 83982/120000 [00:00&lt;00:00, 313940.89 examples/s]"
          }
        },
        "14676964e80b401e9f73d7e46d1d2f3e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "d75d6bd5104b44aaa88b3e3bdc11d884": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8a0b7f8c73944a7d9abfcfd5525344fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "acaaa7ad4ee4409f894dae6a32ae444e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85dbc689123540e58a343905eae2aec0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b2514562cade4be982537f0e3eaa06ae": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "54eafeb7352341238077208abc56a753": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c3a6f01347494d0ab6744a9e878cabfa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bd02e7658a0e46a1aecc8fc959ffc44c",
              "IPY_MODEL_4590d63e2bfa4df5b99e54623ccbb4f6",
              "IPY_MODEL_8cfde95e14fe4181a456ff150e8a09cf"
            ],
            "layout": "IPY_MODEL_bbbbceb91a434da591df82e17696efdf"
          }
        },
        "bd02e7658a0e46a1aecc8fc959ffc44c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8532a6042986455eb202e0dc027df34c",
            "placeholder": "​",
            "style": "IPY_MODEL_08a74f470b7f43dcb37cd8ddb6f02a8d",
            "value": "Generating test examples...:  97%"
          }
        },
        "4590d63e2bfa4df5b99e54623ccbb4f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e1b40d82557c4362babea2699a9eac5c",
            "max": 7600,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9f9b72704bfa4294aac194062c0b71e4",
            "value": 7600
          }
        },
        "8cfde95e14fe4181a456ff150e8a09cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e6954eb93d24a91a35b14f5865492f7",
            "placeholder": "​",
            "style": "IPY_MODEL_1e404b3f054547af875603e31a936f29",
            "value": " 7351/7600 [00:00&lt;00:00, 8169.14 examples/s]"
          }
        },
        "bbbbceb91a434da591df82e17696efdf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "8532a6042986455eb202e0dc027df34c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "08a74f470b7f43dcb37cd8ddb6f02a8d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e1b40d82557c4362babea2699a9eac5c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9f9b72704bfa4294aac194062c0b71e4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3e6954eb93d24a91a35b14f5865492f7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1e404b3f054547af875603e31a936f29": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c8d130cea21d47359afb5c7b84a74ef1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_67fd73d20c2b46e8b7967d42bc49129b",
              "IPY_MODEL_a169a9583c24471f8b395aadc7a62b92",
              "IPY_MODEL_78130d3bb3bc4c48b52f229113b77a62"
            ],
            "layout": "IPY_MODEL_1958fdfb3407464cb4aeba87cd581466"
          }
        },
        "67fd73d20c2b46e8b7967d42bc49129b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6fbe75168d5d4091919116209bc1be74",
            "placeholder": "​",
            "style": "IPY_MODEL_a1efa58385ac4978b32468398c1b7c71",
            "value": "Shuffling ~/tensorflow_datasets/ag_news_subset/1.0.0.incompleteS3CHIE/ag_news_subset-test.tfrecord*...:   0%"
          }
        },
        "a169a9583c24471f8b395aadc7a62b92": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1150c0c0c59947fcaf3ebc98ecee7ada",
            "max": 7600,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a749bf6f121b499399887d6800af56ed",
            "value": 7600
          }
        },
        "78130d3bb3bc4c48b52f229113b77a62": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_55fcc48243474f5894ea2841668544bb",
            "placeholder": "​",
            "style": "IPY_MODEL_002a2a020344451ab1e251b28788a06f",
            "value": " 0/7600 [00:00&lt;?, ? examples/s]"
          }
        },
        "1958fdfb3407464cb4aeba87cd581466": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "6fbe75168d5d4091919116209bc1be74": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a1efa58385ac4978b32468398c1b7c71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1150c0c0c59947fcaf3ebc98ecee7ada": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a749bf6f121b499399887d6800af56ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "55fcc48243474f5894ea2841668544bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "002a2a020344451ab1e251b28788a06f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}